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HUMINT EVALUATION SEMINAR 


Summary 
A group of 27 people from throughout the intelligence 


community met some months ago under the auspices of the 
Center for the Study of Intelligence to share information 
and ideas on methods being used in evaluating human source 
intelligence (HUMINT). The purpose was to broaden the 
awareness and understanding of current HUMINT evaluation 
activities and of their usefulness, and to identify addi- 
tional techniques which might be applied to enhance their 
utility. | 

Two days of frank discussion by the group -- bcoediy 
representative of the views of HUMINT collection managers, 
intelligence and budget analysts, planners, production 
coordinators, consumers, and overseers -- led to the 
consensus that current HUMINT evaluation efforts vary in 
utility and tend to focus largely on the quality of reporting. 
It was felt that more emphasis should and could be placed 
on measuring the efficiency and effectiveness of HUMINT 
programs and of other intelligence functions as well. There 
was agreement on a number of recommendations that would 
increase the usefulness of current HUMINT evaluation efforts, 
and foster the application of better evaluation methods by 


HUMINT managers. It was also agreed that further inter-agency 


discussions of the subject could be fruitful. 
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Conclusions 

The competition for resources among intelligence programs 
has been heightened by the growing centralization of community 
management and budget activities, combined with stringent 
dollar ceilings in a period of inflation. 

Sound decisions on resource allocations, including those 
affecting HUMINT programs, would be facilitated by the seis 
ability of data which measure the relative value or effec- 
tiveness of these programs. 

CIA, DIA, and State have each developed HUMINT evalua- 
tion efforts designed to meet limited agency needs; each of 
these has merit, but they have been uncoordinated and there 
has been little sharing of experience or information on 
evaluation systems among agencies. 

With few exceptions, current evaluation efforts focus 
on measuring the quality of the product, with little emphasis 
on measuring the efficiency of programs (i.e. comparing 
relationships between products and the resources expended 
in their production), or in measuring program effectiveness 
(i.e. comparing results against objectives, or comparing the 
value of results or effects of the program with the value 
of the resources expended). 

Management techniques and systems are available which 
could be applied or adapted by intelligence community components 
to facilitate the development of more useful measures of the 


effectiveness of HUMINT programs than those now being employed. 
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Recommendations 
There should be established or designated a focal point 
in the intelligence community for fostering improved HUMINT 
evaluation activities. This function could be performed 
usefully in the HUMINT element of the Collection Tasking 
Staff. | 
This facilitating element could perform the following 
functions in support of community components: 
+Develop and neeteaan an up-to-date data base 
on individual HUMINT evaluation efforts within the 
community, and track those activities on a continuing 
basis. 
+Advise and assist community HUMINT components 
on evaluation methods, sponsoring workshops and 
seminars on the subject as appropriate. 
+Sponsor and oversee a program of applied research 
on program evaluation techniques, to adapt state-of-the- 
art management evaluation methods to intelligence 
community needs. Such applied research activity could 
be coordinated with CIA's Office of Research and 
Development. 
+Serve as a link with other community elements 
engaged in program evaluation activities (other than 
HUMINT), and with community elements concerned with 


management decisions on resource allocation. 


3 
Approved For Releas€ pOQ4M ab ERARpPyamqo" 71R001500160005-0 


Approved S Rele (Sq 3qnp Fa f EPPA 1R001500160005-0 2 


Community components, either independently or collec- 
tively in the context of a coordinated community effort, 
should consider experimenting with and testing techniques 
such as the following, with a view to improving the utility 
of present HUMINT evaluation efforts: 

+Develop descriptive models of the HUMINT program 
being evaluated, including such details as flow charts, 
linear responsibility charts, and possibly simulation 
models, to facilitate analysis of the program. 

+Analyze the program (or sub-system) to define 

its goals and objectives, identify the nature of the 

reporting requirement which the evaluation is to 

satisfy, and determine the appropriate kind and 

level of evaluation (i.e. quality, efficiency, or 

effectiveness), and identify points at which 

measurements should be made. 

+Develop appropriate measures, define the 

attributes to be measured, and devise ways of collecting 

and presenting the data required to permit the measurement. 

+Identify and document points where the product or 
results of the program have impact, and devise means for 
assessing that impact. 

Individual efforts to improve HUMINT evaluation techniques 


should proceed on their own merit; those HUMINT evaluation 
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activities which prove useful are likely to find a place 
within any overall community evaluation systems which may 


ultimately evolve. 
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Introduction 

A particularly vexing problem for the intelligence 
community in this period of structural transition, budgetary 
constraint, and inflationary costs, is that of deciding among 
competing demands for increasingly scarce resources. The new 
focus on community approaches and the development of new 
institutional forms for establishing intelligence requirements 
and priorities, for tasking collection agencies, and for 
managing resources, have altered traditional budgetary con- 
cepts and procedures, and intensified the competition for 
resources. These conditions have created new interest in the 
question of how one measures the value of an intelligence 
activity -- how one determines whether the results of any 
part of the intelligence process are worth the resources 
devoted to it. 

In one sense, at least, the concept of "evaluation" is 
inherent in the intelligence process, and a variety of 
approaches to evaluating the process and product of intelli- 
gence have been tried over the years. The intelligence 
analyst -- consciously or unconsciously, systematically or 
by intuition -- "evaluates" every piece of intelligence 
information that comes to his attention as he processes his 
"take." The agent handler or defense attache "evaluates" the 
information in determining what to include in his field report. 
Collectors are interested in establishing -- through feedback -- 


the relative quality of a specific report, or of a stream of 
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reporting from a particular source. Collection managers 

and program analysts are concerned with the relative value 

or utility -- in both a quantitative and qualitative sense -- 
of a reporting unit or field station. Others are concerned 
with the proportional contribution of various kinds of collec- 
tors to finished intelligence products. Budget officers, 
resource managers, and legislators are concerned with the 
costs of various programs in relation to the benefits derived 
therefrom. 

Techniques used in making these different kinds of evalua- 
tion range from intuition to sophisticated and complex com- 
puterized models. The latter are more commonly associated 
with technical collection systems, rather than with HUMINT 
collection. One group of CIA HUMINT collection managers has 
evolved a fairly comprehensive evaluation system based on 
consumer feedback, and wants to develop it further. They 
felt a need to share their experience with others, and to 
learn what others are doing in this area. Of particular 
concern to them was the problem of usefully measuring the 
impact of HUMINT reporting. 

These concerns led to this seminar on HUMINT evaluation. 
The purpose of the seminar was to permit a broadly representa- 
tive group of persons concerned with the issue to exchange 
information on HUMINT evaluation as a process. The objective 


was to broaden the understanding of all participants concerning 
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the procedures and methods being used in various agencies for 
evaluating HUMINT, and to arrive at an appreciation of the 
utility of these methods. It was also hoped that -- with the 
aid of an outside management consultant specializing in 
program assessment activities -- the group could explore the 
availability of other evaluation methods and techniques which 
might improve the quality of current evaluation efforts or 
permit the development of new systems. 

To achieve these aims, the Center for the Study of 
Intelligence assembled a group with varying backgrounds, but 
sharing a common interest in evaluating HUMINT in one fashion 
or another. The composition of the group was such as to give 
some expression to the needs and concerns of legislators, 
policymakers, resource managers, budget officers, collection 
managers, intelligence analysts, and R&D managers. The agenda 
provided for: 

-- discussion of the evaluation interests and concerns 
of those present; 

-- briefings on HUMINT evaluation practices in the 
Operations Directorate of CIA, and in NFAC, State, and DIA; 

-- consideration of evaluation techniques used in other 
government agencies and in the business world, and discussion 
of how these might affect our needs; and 

-- a determination of further activities that could be 


undertaken as a follow-on to such a seminar. 
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General Discussion 

The discussion opened with consideration of the broader 
challenge presented to the intelligence community by the 
Policy Review Committee on Intelligence (PRC/I) in terms of 
the need to institutionalize feedback to the guidance provided 
in the form of National Intelligence Topics (NITs). It was 
observed that some means must be developed to measure (evaluate) 
the community's responsiveness to the NITs, and that ultimately 
that response will have to be tied in some way to decisions on 
budget and resource allocations. It was acknowledged that we 
are a long way from being able to do that, but the view was 
offered that the community's progress in this area has thus 
far been sluggish, although DIA's KIR program is a step in the 
right direction. 

The NITs came in for considerable discussion. While it 
was generally recognized that they provide useful guidance 
regarding the interests and needs of top policymakers, it 
was felt that they have limited utility in the context of 
program evaluation. The nature of the NIT "questions" 
precludes simple yes or no answers. The concept also poses 
problems concerning the adequacy of responses -- How much 
information is "enough"? How much of the information needed 
can be dealt with given the resources available? The consensus ~ 
leaned toward the view that an effective evaluation system 


for intelligence activities required a more clearcut statement 
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of objectives than was provided by the NITs. Objectives which 
would drive the kinds of performance evaluation the group felt 
were desirable needed to be more precise, definable, and 
quantifiable, and less open-ended than the NITs. 

There was substantial discussion of the peculiar vulner- 
ability of HUMINT in the competition for resources. The 
collection potential of technical systems is easily measured; 
hardware specifications can be designed to ensure the desired 
coverage capabilities; research and development programs are 
readily costed, and lead times between initial investment and 
operational availability can be determined with some reli- 
ability; performance of the system can be meaningfully 
measured in terms of target coverage. By contrast, HUMINT 
sources defy specific programming; long-term investment risks 
and costs are difficult to project and justify; target 
coverage cannot be predicted with confidence. Because HUMINT 
is a labor-intensive system -- as opposed to the capital-inten- 
sive technical systems -- HUMINT is harder hit in the near- 
term by spiraling manpower costs. 

This vulnerability is most evident in questions raised 
by the recent cut in personnel of CIA's Operations Directorate. 
Congressional staffers asked what the impact of a cut of that 
size would be, in contrast to a cut of greater or lesser size. 
If 820, why not 1,020, or 620? What difference would a cut 


of different size make? This tied in again with the question 
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raised above -- in the intelligence business, how much is 
enough? When do we need more information, when do we have 
too much? How can this be measured? 

This dilemma reflects the need for a means of evaluating 
the contribution of HUMINT to the intelligence process, and 
its impact on other programs which it affects, either directly 
or indirectly. It was agreed that if HUMINT managers could 
demonstrate the effect of a particular report -- or series of 
reports -- on U.S. military R&D programs, for example, or an 
trade negotiations, the problem of justifying resources for 
HUMINT woud be more manageable. 

There was disagreement on the extent to which this sort 
of impact analysis or evaluation could be carried. How does 
one evaluate the impact of a HUMINT report -- or any other 
intelligence product -- on policy decisions? Some believed 
that it was essential for the intelligence community to develop 
the capability to measure its impact on policy and that it would 
be feasible to do so. Others felt that the nature of the 
intelligence process precluded this; national security policies 
are not conducted on a profit-loss basis, and national objec- 
tives themselves are not defined in ways that facilitate objec-. 
tive measurement of success or failure in the conduct of 
national policy. Still others felt this issue of measuring 


impact on policy was imposing too rigorous a straitjacket ~ 
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on the problem of intelligence evaluation; that intelligence 
was a matter of storing up knowledge, and that whether and how 
the policymaker used that knowledge was beyond the responsi- 
bility of the intelligence community. The tendency to view 
intelligence as a "production" process drew some criticism. 

Analogies were drawn to the military services, and to 
the forest service. These elements are budgeted to establish 
capabilities for performing services (fighting wars, or 
extinguishing forest fires), and are evaluated in "peacetime" 
not in terms of how well they fight wars or put out fires, but 
in terms of their readiness to perform the tasks for which they 
are maintained. Those interested in justifying resource 
expenditures can take credit for indirect benefits or side- 
effects, such as airlift of refugees, or favorable fire- 
prevention results, but these services are not expected to 
justify every dollar spent on resources if they are not called 
upon to perform the role for which they are established. Some 
felt these factors are relevant, to some extent, to the intelli- 
gence community, in terms of long-term investment in collection 
resources which are not expected to produce results today, and 
of maintenance of capabilities for covert action, which may not 
be employed to their full capacity. 

The majority seemed to feel that it may be both feasible 


and appropriate for the intelligence community to demonstrate 
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its impact on U.S. Government programs and policies in specific 
instances. Although it might not be possible to account in 

this fashion for every dollar spent on intelligence activities -- 
and specifically on HUMINT -- it should be possible to account 
for a more ‘Significant portion of intelligence expenditures 

than has hitherto been the case. This turned the discussion 


to consideration of HUMINT evaluation efforts currently under way. 
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Current Evaluation Efforts -- Resources Management Staff 


Charged with responsibility for preparation of the intelli- 
gence community budget, the Resources Management Staff has 
established program evaluation as an essential part of its 
operation. | An indirect technique is employed whereby RM 
personnel draw evaluations from program operators. The points 
of concentration are upon program success or failure, and 
possible duplication of effort. The procedure is designed to 
relate resource consumption to results in support of budget 
proposals. While dependent largely on the cooperation of 
those responsible for management of programs, both "carrot 
and stick" incentives are available to attain the. level of 
candor and detail required. 

To date, evaluation of human source reporting has been 
indirect and limited in depth. In comparative resource 
requirement terms, human source reporting has not been viewed 
as a major expense. Overall it represents about 8 to 9 - 
percent of the budget while providing source material for 
70 percent of the daily "Morning Report" of INR. 

More and better evaluation by the various components 
collecting human source intelligence would benefit signifi- 


cantly the work of the RM staff. 
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CIA--Operations Directorate Evaluation of Field Reporting 


The Evaluation Staff of the Operations Directorate (DO) 
comprises a Foreign Intelligence and Records Unit. A total 
team of 10 persons, of whom only 3 are full time, is used. 

The principal technique is that of direct interviews with 

users of clandestine and domestic collection reporting. Upwards 
of 500 consumers of the reporting are interviewed quarterly, 
each of the 10 staff members averaging calls upon 50 people 
apiece. 

This procedure provides a continuous flow of evatuative 
judgments on quality and usefulness of these reporting elements. 
The results clearly are effective in program management and 
budget qualification terms. 

The complete DO evaluation process begins in the field, 
MS Gy ENS EL 2) SOE Senet Ste asmat a Report sie eiceveluee 
tive judgment. This judgment in turn is based, at least 
partially, on guidance from Washington, which in itself may 
have been derived from evaluation of earlier reporting and 
perceived needs. The link between evaluation and guidance 
is solid, and each tends to strengthen the other. 

Reports are graded under two distinct systems in the DO. 


© Regional divisions grade their own for quality control reasons. 


ai 


At theentral management level,other grading criteria are 


at vere arsenal 


applied{ 28 
K1 [ss centrat management places more stress on the 


user interview results in its drive to maintain usefulness. 
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An electronic data processing system and program pro- 
vides report listings by month. These are the raw material 
sources for the interviewers, who are able to break out 
report topics to match the established areas of interest 
for each interview. In this manner each official interviewed 
is rendering judgments within his area of expertise. 

While this technique does not provide direct cost-benefit 
ratios, it does allow determinations of the usefulness of 
reporting effort or of resources. The synthesis of views ; 
the guidance procedure. 

An effort to evaluate the hard value or dollar impact of 
human source reporting provided examples from the military 
technical scene. Cases were cited where reporting led to 
program changes or cost reductions in weapons design or pro- 
curement. This evaluation effort will receive increased 
attention as it is capable of developing easily demonstrable 
evidence of major benefit.* 

State Department--Evaluation of Embassy Reportings 

From mid-1976 through mid-1978 the management level at the 
Department of State sponsored the development and operation of 
a system to evaluate the quality of substantive reporting from 


Foreign Service Posts. Evaluations were begun on 48 posts, 


with 15 carried completely through the six-step procedure. 


e 
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*§ *A more detailed description of this_assessment program 
gro) was published in Studies in Intelli Oo) ia 
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Recognizing the characteristics of the high-volume, vol- 
untary or largely post-initiated nature of Foreign Service 
reporting, the evaluators worked with a year's samples of 
telegraphic reporting for each post. It was determined 
statistically that for posts transmitting 1,000 or more 
messages per year, a 500 item sample would provide a 95 per- 
cent confidence level of being representative of the total. 
All non-substantive messages were eliminated from the sample, 
leaving usually a range of 40 to 70 messages to evaluate. 

Evaluation of five attributes--source, organization, 
analytical content, relevance and usefulness--was performed 
by panels of six experienced Foreign Service Officers, three 
economic and three of political background. The message 
sample separated economic and political material, which was 
read and scored for each attribute by the appropriate panelists. 

Scoring results were used as guides to discussions with 
State officers of the country office responsible for the 
particular posts. Following these reviews, questions were 
formulated for inspection teams enroute to the post. Following 
a debriefing of the inspectors on their return, a memorandum 
was drafted and reviewed with the country office. Thereafter, 
a final memorandum was addressed to the Chief of Mission of 
the post, and the Assistant Secretary in charge of the geo- 


graphic bureau. 
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This evaluation procedure was fengthy and cautious, 
inexpensive in use of resources, and of limited success. — It 
showed that such a system can provide worthwhile results. It 
sharpened interest in the subject of reporting evaluation 
per se, and the particular attributes upon which it focused. 
Its principal achievement was realization in State of the 
need for improved guidance to Foreign Service posts by their 
country offices, leading to the recent adoption of the 
"reporting plan" system. The evaluation system failed to 
establish a link between reporting quality or usefulness 
and the use of resources in a manner helpful .to State 
management, and the program was moribund in late 1978. 
DIA--Field Report Evaluation 

Several forms of information, report or collector evalua- 
tion are utilized by DIA. Of principal interest to the 
seminar is the effort devoted to field report evaluation as 
opposed to the extensive efforts employed in DIA output or 
production evaluation (see below). 


A sampling system based on an EDP storage system has 


25X1 


can be manipulated by source to help the collector determine 


the value and reliability of a source; by geographic area, 
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to ensure reasonable coverage of posts; and by product, to 
examine the extent of source material available to meet 
a pre-determined need. 

The IR sampling system provides a useful check on 
overall quality. It does not allow complete topical or sub- 
ject evaluation, but does assist the development of guidance 
to the field. It appears to be of primary interest to the 
producers of finished intelligence products--intelligence 
summaries, estimates, etc. 

DIA participants suggested that to evaluate in terms of 
effectiveness would require comparisons of finished products 
derived from human source intelligence or reporting against 
their contribution toward stated goals, such as preparation 
of national estimates, military capability studies, daily or 
periodic reviews. The results of such comparisons could 
influence decisions affecting resource allocation. 
DIA--Other Evaluation Programs 

As a consumer/user of human source information, DIA 
evaluates field reports of other agencies, passing the results 
to the supplying organization, primarily to CIA's Operations 
Directorate. 

Direct performance evaluation is applied to the Defense 
Attache system through periodic review of the output of a 
“aanated number of attache posts by the DIA Director. This 


is a composite evaluation system incorporating the views of 
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DIA users of attache output. Guidance provided the Defense 
Attaches is based on the results of this performance evaluation. 
Similarly, through participation in the National Intelli- 
gence Tasking Office evaluations, DIA has been able to reflect 
its views of needs on issues or topics affecting the full 
range of collection systems, derived from its own surveys 
and evaluation of information recéived or available. 
DIA--Production Evaluation 

DIA is interested in the development of an evaluation 
system that would cover all aspects of the flow of information 
from the input of raw material through the output of dissemin- 
ated final product. 

Through an established program, DIA periodically solicits 
consumer evaluations of recurring products in an effort to 
gain a comparative view and spot developing trends. In 
addition, it asks each DOD general intelligence production 
organization to conduct surveys of its own output for internal 
guidance and for the use of DIA's Project Management Office. 

In order to better anticipate or meet the needs of its 
wide range of consumers, DIA is developing a record of all 
evaluation efforts, including the office conducting the 
survey, the purpose, questions posed, and the sample popula- 
tion. It is expected that this record will improve the 


tailoring of products to meet the needs of individual 
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consumers and facilitate monitoring changes in the consumer 


judgments about product utility. 
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Assessment of Community Efforts 


a Seminar consultant on evaluation metho- 


dology, picked up on the brief description that was given of 
the forthcoming National HUMINT Tasking System. He saw the 
tasking of HUMINT collectors with national requirements under 
this system as probably helpful in better defining their objec- 
tives against the goals represented by broad, all-source National 
Intelligence Topics. = such definition would 25X1 
help in improving the HUMINT reporting evaluation systems des- 
cribed by the participants. He characterized all of these 
systems, however, as being too low in the general hierarchy of 
possible evaluation systems. He saw almost exclusive focus 
in the CIA, DIA and State Department systems upon quality 
measurement, as opposed to that which OMB has clearly been 
calling for in the way of efficiency and effectiveness assess- 
ment. Passing out copies of extracts from certain 1978 OMB 
publications,[___—i|drew the group's attention particularly 
to the following passages pertaining to the zero-base budgeting 
process: 

"Meaningful objectives should be established for 

each decision unit. Major objectives should be 

explicit statements of intended output clearly 


relative to the mission of the program or organization." 
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"The expected results of work performed or services 
provided should be identified to the maximum extent 
possible through the use of quantitative measures." 

"The assessment of alternatives should be based on 

the relative effectiveness and efficiency of each 

alternative in accomplishing major objectives." 

"To the extent possible, using specific measures 

of accomplishment, workload, effectiveness and 

efficiency, describe the recent progress of the 

decision unit in achieving its objectives. The 

key indicators by which outputs and achievements 

will be measured should be obtainable from existing 

evaluation systems... Indirect or proxy measures 

should be used only while evaluation and work measure- 
ment systems are being developed.''* 

Explaining his reference to the hierarchy of possible 
evaluation systems] went on to equate evaluation 
with assessment and to lay out a model (Attachment A). He 
related it to HUMINT, and used the model to focus the dis- 
cussion on the types of evaluation of HUMINT called for by 


OMB. He used the following terms in explanations: 


*OMB Circular No. A-11 "Preparation and Submission of 
Budget Estimates." May 25, 1978. No. A-115. ''Zero-base 
Budgeting."' May 5, 1978. 
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Evaluation of Quality 


This involves identifying the attributes that a HUMINT 
report has. The more objective the attributes, the more 
reliable any judgments as to quality. As the existence of 
quality is determined, so it is established that meaningful 
output has been produced. The more specific, measureable 
and quantifiable the attributes of the output (a HUMINT 
report) the more reliable its quality measurement. 


Evaluation of Efficiency 


This requires determination of the relationship between 
HUMINT output and the effort that went into its production. 
Establishing a productivity index relates any meaningful 
measure of output to any meaningful measure of input to it. 
Going beyond that, work measurement relates the human 
resources (hours, months, years) consumed in the production 
of output to the output itself. Determining unit-cost 
efficiency calls for relating the cost of all accountable 


and allocatable resources to a unit of output produced. 


Evaluation of Effectiveness 


This requires determination of the results or effects i 
of HUMINT output and relating them to input. Results may 
be measured in terms of stated objectives or in the same 


value terms as the input. The former allows effectiveness 
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assessment to take place. The latter permits benefit-cost 
(or cost-benefit) analysis. 

[ _|saia he foresaw increasing pressures on the 
intelligence community from OMB and Congress to engage in 
both efficiency and effectiveness assessment of its HUMINT 
collection. Citing examples from the experience of other 
federal agencies in this regard, he made several key points: 

Efficiency and effectiveness measurement both require 
relating output to input, but past output can be used con- 
vincingly to justify future input costs. There may be no 
requirement for making a direct correlation between certain 
output and certain input. It will likely be sufficient if 
it can be demonstrated that there is cause to believe that 
a relationship exists. 

Moving beyond efficiency measurement to the assessment 
of cost effectiveness requires clear definition of output 
in terms of results achieved, but they do not need to carry 
a value. 

Going the final step, to benefit-cost analysis, does 
require placement of a value upon output results, but it 
can be imputed value and does not require quantification 
beyond the use of "shadow prices.” 


[_—iexplained "shadow prices" as an indication of 


the worth of output results. They should not be taken as 
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a definitive and absolute measure, but rather as a dynamic 
working statement of the value or benefit of the results, 
as derived in three major areas: 


Market Value - The price for the results as they 
might be incurred in market transactions. 


Cost Reductions/Avoidances - The currently borne 
social, political, economic or military costs/ 


expenditures that would be reduced or avoided 
by the results/effects (of HUMINT reporting). 


Indirect, Intangible Value - Examples: inde- 

pendence, security, peace, cultural identity, 

quality of life, etc. 

| —_ieseripeion of different levels in the hierarchy 
of evaluation systems facilitated discussion of increasing 
centralization in intelligence management. More emphasis 
upon quantification of response to objectives and the value 
of program results was seen to bring inevitably more need 
for improvement in evaluation methodology and techniques. 
Current, largely disjointed efforts in this regard would 
benefit from some coordination, according to[ _Jand 25X1 
unnecessary duplication of experimental effort could be 
reduced. He mentioned the potential benefits to all of a 
central repository of impact data that could be drawn upon 
for evaluation purposes, along with the results of applied 
research in intelligence evaluation.[ _|stressed the 25X11 
multitude of tools available for evaluation in general 


and the need simply to adapt these to the intelligence pro- 


fession. He described the usefulness of linear responsibility 
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charting in matching evaluation efforts to actual need for 
them and in identifying the kinds of impact that collection 
results are supposed to have, and are likely to have. 

This led to a discussion of shortfalls in HUMINT 
reporting and the need to communicate this upward when it 
occurs. 

Informalization or shortfall is the hallmark of a good 
system, according [ ana the ability to demonstrate 
value tradeoff associated with various collection activities 
should be put to use with OMB. More frankness about that 
which simply cannot be done cost-effectively, with favorable 
benefit/cost ratio, would be consistent with the stretching 
of resources now going on in the face of rising demands for 
intelligence information. 

An important step in better rationalization of resources 
is the development of reliable resources for moving beyond 
quality measurement to the assessment of efficiency and 


effectiveness in HUMINT reporting. 
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ANNEX "A" 
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ANNEX "A" 


Quality 


Inputs Outputs 
ace > 
Process 
F 
($, People, 7 (Goods, 
resources) Z Services) 


e Efficiency 
\ 
\ 
v 
N 7 Results/Effects 


Effectiveness 
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Quality 
- Definition of Output 


- Attributes of Output 


Establishes that we have an output of 
at least minimally acceptable quality 


(vs. stated attributes). 
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Efficiency 


- Work Measurement 
- Unit Cost 


- Productivity Index 


Is a relative measure -- allows for 
ee 
comparison to standard, history, or 


other operation. 
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Effectiveness 


- Cost Effectiveness 


- Benefit-Cost 


Both require "Proof" of outputs 
having a result or effect. Allows 


for evaluation of alternatives. 
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Objectives 
Two Classes: 
- MUST = absolute, limit 
- WANTS = trade-off, range 
Two Formats: 
- Upper Bound 


Lower Bound 


General 
- Must be specific, measurable 
Usually in pairs-MUST & WANT 
Always a limit 
Needs focus-Input, Output, 
Result 
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Relative Efficiency 


WM: 
Eff = Earned Hrs./Active Hrs. 
Earned Hrs = Units Output 


X Std. Rate 


uc: 
Eff = Std. Cost/Unit 
Actual Cost/Unit 


Eff 


Outputs/Inputs 
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Effectiveness 
lst Definition 


Effect. 


Results/Objectives 
2nd Definition 


Effect. = Results/Resources 


Note: 
Use of the Ist Definition presumes 
that the activity was effective from 


2nd Definition viewpoint. 
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Cost Effectiveness 
- Fixed-Cost, max. effect. 


- Fixed-Effect., min. cost 


Note: 
- Both require a measure of 


effect. 


- Assumes result or effect 


worth resources used. 


- Needs "RCTB" outputs caused 


results. 
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Establishing Causal Link 
Ground Rules 


Positive-constructive-competitive 
Scientific investigation 


"RCTB" vs. "Proof" 


a7. 
Approved For RA NF/ BEN TTA meeeotsoons® 


Approved @ CONFIDEN TAP ow R001500160005-0 


- Some Approaches 
Content Analysis 
Records Search 
Interviews, Questionnaires 
Quasi-experimental: 
- time series 
- multiple time series 
Non-experimental: 
- before vs. after 
- after only 


(comparison) 
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Benefit-Cost 


Addresses "value" question - 
are results worth resources 


expended? 


Requires derivation of 
"shadow prices" (both 


resources and results). 


An approach based on economic 


analysis and.evaluation. 
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Shadow Prices 

= ‘*Notational values for evaluation 
and analysis purposes. 

= Dynamic-situational and subjective. 

= "Social" viewpoint=general national, 
social, public interest. 

= Areas of derivation: 
-"Market" 
-Cost reduction/avoidance 
-Intangibles 

= Positive-constructive-competitive 
viewpoint. 


= RCTB 
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OUTPUTS 


INPUTS 
Seren wescol Memes” 
(Goods, 


($, people, 
resources) services) 


v 
Results/Effects 
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